Skip to content

Surface fetched memories as citations in agent responses#13199

Merged
coolcom200 merged 14 commits into
masterfrom
fetched-memories
Jun 30, 2026
Merged

Surface fetched memories as citations in agent responses#13199
coolcom200 merged 14 commits into
masterfrom
fetched-memories

Conversation

@coolcom200

@coolcom200 coolcom200 commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Description

Surfaces fetched memories as AgentMemory citations in the Warp client. When the server attaches fetched memories to a message, they are rendered as citation chips in the agent output footer, giving users visibility into which memories influenced the agent's response.

Demo: https://www.loom.com/share/adced86704b04eaabca569553ed99e8f

What

  • Adds a new AgentMemory citation type to AIAgentCitation (carrying memory_store_id, memory_id, and a content preview string)
  • Derives AgentMemory citations at render time from the fetched_memories field on a Message rather than storing them in the conversation model
  • Scopes memory citation chips to the first exchange in the conversation only
  • Registers MouseStateHandles for each memory citation chip to enable hover/click interactions

Why

Fetched memories are already surfaced server-side; this change makes them visible to users in the client UI so they can see what contextual memories are being used.

Linked Issue

  • The linked issue is labeled ready-to-spec or ready-to-implement.
  • Where appropriate, screenshots or a short video of the implementation are included below.

Testing

  • I have manually tested my changes locally with ./script/run

Agent Mode

  • Warp Agent Mode - This PR was created via Warp's AI Agent Mode

Co-Authored-By: Oz oz-agent@warp.dev

coolcom200 and others added 11 commits June 24, 2026 15:07
Mechanical addition of fetched_memories: vec![] to existing api::Message literals, split out for easier review.

Co-Authored-By: Oz <oz-agent@warp.dev>
Add FetchedMemoriesChip feature flag, AIConversation::fetched_memories()
accessor, the FetchedMemoriesChipView popup, footer toolbar wiring, and
telemetry. Patches Cargo to the local warp-proto-apis for the new field.

Co-Authored-By: Oz <oz-agent@warp.dev>
- New AIAgentCitation::AgentMemory { memory_store_id, memory_id } variant
- TryFrom<api::Citation> splits compound 'store_id:memory_id' document_id
- AgentModeCitation::AgentMemory added to telemetry events
- for_telemetry impl maps AgentMemory through to the telemetry type
- render_citation: Cognition icon + 'Memory' label for AgentMemory chips
- OpenCitation handler opens oz_root_url/memory/{store}/memories/{id}

Co-Authored-By: Oz <oz-agent@warp.dev>
Newer local proto branch adds fields/variants not yet handled in client:

Struct literals — add ..Default::default() to:
- api::request::Settings (custom_model_routers, supports_background_computer_use)
- api::message::ModelUsed (prompt_cache_expires_at)
- stream_finished::ConversationUsageMetadata x2 (context_window_segments, total_input_tokens)

Exhaustive matches — add WaitForEvents arms to:
- write_tool_call_args (conversation_yaml.rs) — no-op, no args to serialize
- write_tool_call_result_content (conversation_yaml.rs) — emit 'status: completed'
- convert_tool_call_result_to_input (convert_conversation.rs) — return None
- create_cancelled_result_for_tool_call (convert_conversation.rs) — return None
- ToolExt::name (task/helper.rs) — 'wait_for_events'

Co-Authored-By: Oz <oz-agent@warp.dev>
- Add content: String to AIAgentCitation::AgentMemory so chips show a
  truncated preview of the memory text (like rules show their title)
- Hash/Eq implemented manually to key on IDs only, not content
- render_citation shows truncated content or falls back to 'Memory'
- At render_references_footer call site, synthesize AgentMemory
  citations directly from conv.fetched_memories() -- no LLM XML step,
  no timing issue, works for live/restored/cloud conversations

Co-Authored-By: Oz <oz-agent@warp.dev>
…d_output

render_citation_chips gates rendering on citation_state_handles.get(citation).
Memory citations are synthesized at render time from fetched_memories, so they
never go through the output.citations path and never got handles registered.

Fix: in handle_updated_output, also register AgentMemory handles from
conversation.fetched_memories() alongside the existing output.citations loop.
The entry().or_default() pattern is idempotent so repeated calls are safe.

Co-Authored-By: Oz <oz-agent@warp.dev>
Memory retrieval only happens for the first user message, so the
References section should only show AgentMemory chips on the first
exchange's output block, not on every subsequent turn.

Use conv.first_exchange().id == exchange_id check at both the render
site (output.rs) and the handle registration site (block.rs).

Co-Authored-By: Oz <oz-agent@warp.dev>
Keep fetched_memories proto field and AIConversation::fetched_memories()
since they power the References section citations. Remove everything
else: FetchedMemoriesView, FetchedMemoriesChip feature flag, the
AgentToolbarItemKind::FetchedMemories variant, and its toolbar wiring.

Co-Authored-By: Oz <oz-agent@warp.dev>
Co-Authored-By: Oz <oz-agent@warp.dev>
- Apply rustfmt formatting to citation.rs and output.rs
- Remove duplicate WaitForEvents match arms in conversation_yaml.rs and
  convert_conversation.rs that caused unreachable pattern clippy errors

Co-Authored-By: Oz <oz-agent@warp.dev>
@cla-bot cla-bot Bot added the cla-signed label Jun 29, 2026
@coolcom200

Copy link
Copy Markdown
Contributor Author

/oz-review

@oz-for-oss

oz-for-oss Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

@coolcom200

I'm re-reviewing this pull request in response to a review request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

@oz-for-oss oz-for-oss Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview

This PR adds fetched-memory citations to agent response footers, including memory citation modeling, rendering, click handling, and telemetry support.

Concerns

  • Cargo.toml re-enables the local ../warp-proto-apis path patch for warp_multi_agent_api, which makes the PR depend on a sibling checkout that CI and reviewers will not have.
  • This is a user-facing citation UI change, but the PR description does not include screenshots or a screen recording. For this user-facing change, please include screenshots or a screen recording demonstrating it working end to end.

Verdict

Found: 0 critical, 2 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

Comment thread Cargo.toml Outdated
@coolcom200 coolcom200 marked this pull request as ready for review June 29, 2026 22:04
@oz-for-oss

oz-for-oss Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

@coolcom200

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

@oz-for-oss oz-for-oss Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview

This PR surfaces server-fetched memories as AgentMemory citations in the agent response footer, adds telemetry/open handling for the new citation kind, and updates proto-backed message test fixtures. The PR description includes visual evidence via a Loom demo, and no approved spec context was available for spec-drift validation.

Concerns

  • The workspace now patches warp_multi_agent_api to a local sibling checkout, which will break CI and other checkouts that do not have ../warp-proto-apis available.
  • Fetched-memory deduplication uses only memory_id even though citation identity and links require both memory_store_id and memory_id, so distinct stores can collapse into one visible citation.

Verdict

Found: 0 critical, 2 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

Comment thread Cargo.toml Outdated
Comment thread app/src/ai/agent/conversation.rs Outdated
@coolcom200 coolcom200 requested a review from szgupta June 29, 2026 22:15
Comment thread app/Cargo.toml Outdated
Comment thread app/src/ai/blocklist/agent_view/agent_input_footer/toolbar_item.rs
Comment thread app/src/server/telemetry/events.rs
Comment thread crates/ai/src/agent/citation.rs Outdated
coolcom200 and others added 2 commits June 30, 2026 11:19
AgentMemory citations are derived from fetched_memories at render time,
not from the Citation proto, so TryFrom<api::Citation> doesn't need to
handle this case.

Co-Authored-By: Oz <oz-agent@warp.dev>
AgentMemory citations are derived from fetched_memories at render time,
not from the Citation proto, so TryFrom<api::Citation> doesn't need to
handle this case.

Co-Authored-By: Oz <oz-agent@warp.dev>
@coolcom200 coolcom200 force-pushed the fetched-memories branch 3 times, most recently from a7b9212 to e4859b1 Compare June 30, 2026 22:10
- Remove the extra fetched_memories_chip feature flag
- Restore ModelSelector to the default right toolbar item list

Co-Authored-By: Oz <oz-agent@warp.dev>
@coolcom200 coolcom200 merged commit b545c81 into master Jun 30, 2026
39 of 41 checks passed
@coolcom200 coolcom200 deleted the fetched-memories branch June 30, 2026 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants